Skip to content

Feat/implement phase f#5

Merged
nixrajput merged 18 commits into
mainfrom
feat/implement-phase-f
Jun 21, 2026
Merged

Feat/implement phase f#5
nixrajput merged 18 commits into
mainfrom
feat/implement-phase-f

Conversation

@nixrajput

@nixrajput nixrajput commented Jun 21, 2026

Copy link
Copy Markdown
Owner

Phase F — Advanced Transfer (machinery; several CLI paths gated as honest follow-ups)

Lands the advanced-transfer machinery from spec §9. By design, the working user-facing pieces ship now; the paths that need a live replication-configured DB or driver-surface changes are gated with clear errors and documented as follow-ups (this box has no Docker, so those paths are compile-checked locally and validated by CI). README marks Phase F 🟡 Partial, not ✅.

✅ Works today (real, tested)

  • Dump envelope — every dump is prefixed with a 4 KB SIPH JSON header (type, base/parent IDs, WAL/binlog positions, checksum). Checksum covers envelope+body, so verify stays correct; restore strips it before handing native bytes to the driver.
  • Chain-walking restorerestore <id> resolves the base→incremental chain and applies it in order; --up-to <id> stops early; cycle + broken-chain detection (termination proven).
  • Bounded-buffer streaming syncsync streams through a jobs.Stream (default 64×1 MB) instead of io.Pipe, exposing a FillPercent backpressure metric and propagating a backup failure to the restore side via CloseErr (no truncated dump committed as clean). Race-clean; the no-goroutine-leak invariant is mutation-tested.

🟡 Machinery in place, CLI gated (follow-up wiring)

  • Incremental backup capture — Postgres WAL replication-slot + LSN capture; MySQL/MariaDB binlog-position capture (on the shared mysqlcommon.Conn). backup --incremental returns a clear "not yet wired" error — the end-to-end capture→envelope→catalog wiring + orphan-slot cleanup are tracked follow-ups.
  • Cross-engine sync — canonical schema + MapToNative type matrix + JSONL emit/consume with per-engine identifier quoting and placeholders (unit-tested, incl. SQL-injection guards). sync --cross-engine is capability-gated off until driver.Inspect carries column types (typed schema introspection is the prerequisite).
  • CDC — state-file persistence + resume + capability gating; RunCDC is a polling scaffold (real logical-replication streaming via pglogrepl/binlog tailing is deferred). Gated off (no driver advertises CapCDC); there is no siphon cdc command yet.

Notable decisions / deviations from the original plan

  • jobs.Stream design — the plan's Close() closed the buffer channel, which races a concurrent Write (caught by go test -race); the buffer channel is never closed (only a done signal), making Close safe from any goroutine. Added CloseErr (the bounded-buffer analogue of io.PipeWriter.CloseWithError).
  • Cross-engine SQL safety — the plan interpolated identifiers raw (injection vector); fixed with per-engine identifier quoting ("/` doubling) + per-engine placeholders ($n vs ?), with values always parameterized. SQL extracted into pure builders + injection-guard tests.
  • binlog machinery on the shared mysqlcommon.Conn (not per-package Conn types, which don't exist post-Phase-E), parameterized by a binlogBinary name.
  • Honest gatingbackup --incremental, sync --cross-engine, sync --continuous all return clear CodeUser/ErrDriverUnsupported errors rather than silently no-op'ing or falsely succeeding.

Verification

  • make tidy clean · make test (14 packages, no failures) · go test -race ./internal/app/... ./internal/jobs/... clean · make lint 0 (default + --build-tags integration) · make build ok · go build -tags integration ./... compiles.
  • Integration suites (Postgres/MySQL/MariaDB) compile under the integration tag; CI is the first runtime validation of the envelope round-trip + streaming paths against real containers.

Follow-up gates (tracked) before incremental backup ships to users

  • Postgres orphan replication-slot cleanup (a crashed incremental otherwise pins WAL).
  • Validate the pg_receivewal / mysqlbinlog streaming invocations against live wal_level>=replica / binlog_format=ROW servers.
  • Typed schema introspection (column types in driver.Inspect) to unlock cross-engine.

See docs/INCREMENTAL.md, docs/CROSS_ENGINE.md, docs/CDC.md for per-feature status.

Summary by CodeRabbit

Release Notes

  • Documentation

    • Phase F advanced transfer machinery now documented, including incremental backups, cross-engine sync, and change data capture concepts
    • Project status updated to reflect partial Phase F readiness
  • New Features

    • Restore now supports point-in-time recovery within incremental backup chains
    • Incremental backup infrastructure in place (feature-gated, pending final wiring)
    • Cross-engine database sync framework (capability-gated, pending schema introspection)
    • Change data capture continuous mode framework with state persistence (not yet active)

nixrajput added 14 commits June 22, 2026 00:40
Backup writes a dumps.Envelope through the checksum tee before the
driver's native bytes, so the whole-file sha256 (and Verify's rehash)
stays consistent. Restore strips and validates the envelope, handing
the native-bytes body reader to the driver.
…cremental

- Swap Sync's io.Pipe for jobs.Stream(64); use CloseErr to keep
  backup-failure propagation so a truncated dump is not committed.
- Add SyncOpts.CrossEngine/Continuous; route cross-engine through a
  capability-gated runCrossEngineSync that returns a clear unsupported
  error (no driver advertises cross-engine yet).
- Add sync --cross-engine/--continuous and backup --incremental/--base
  flags; continuous and incremental return honest unsupported errors.
- Add Stream CloseErr propagation and clean-EOF unit tests.
@coderabbitai

coderabbitai Bot commented Jun 21, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@nixrajput, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 31 minutes and 11 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more credits in the billing tab to continue.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 976142ef-74c3-40ca-a381-4c1e6241fa1a

📥 Commits

Reviewing files that changed from the base of the PR and between 26f56c7 and d3cbf63.

📒 Files selected for processing (13)
  • README.md
  • docs/CROSS_ENGINE.md
  • internal/app/canonical_consume.go
  • internal/app/canonical_emit.go
  • internal/app/canonical_test.go
  • internal/app/restore.go
  • internal/app/restore_chain_test.go
  • internal/cli/backup.go
  • internal/driver/_mysqlcommon/incremental.go
  • internal/driver/_mysqlcommon/incremental_test.go
  • internal/driver/postgres/incremental.go
  • internal/jobs/stream.go
  • internal/jobs/stream_test.go
📝 Walkthrough

Walkthrough

Phase F adds the core advanced-transfer machinery: a bounded jobs.Stream pipe with backpressure replaces io.Pipe in sync; a 4 KB SIPH JSON envelope wraps every dump; Catalog.ResolveChain reconstructs base→incremental chains; Restore walks chains with --up-to truncation; MySQL/MariaDB binlog and Postgres WAL incremental drivers are implemented; a cross-engine canonical type system with JSONL emit/consume is added; a CDC polling scaffold with state persistence is introduced; CLI gains --incremental, --up-to, --cross-engine, and --continuous flags; and three new concept docs are added.

Changes

Phase F Advanced Transfer

Layer / File(s) Summary
Bounded jobs.Stream pipe
internal/jobs/stream.go, internal/jobs/stream_test.go
Introduces Stream, a bounded channel-based io.Reader/io.Writer/io.Closer with FillPercent backpressure, CloseErr error propagation, overflow buffering, idempotent close, and CloseOnCtx. Tests cover roundtrip, error propagation, partial reads, fill tracking, concurrency safety, and context-driven close.
SIPH envelope format and dump chain resolution
internal/dumps/envelope.go, internal/dumps/chain.go, internal/dumps/envelope_test.go, internal/dumps/chain_test.go
WriteEnvelope/ReadEnvelope implement a fixed 4 KB JSON header with magic prefix and space padding. Catalog.ResolveChain walks ParentID pointers to produce a base→target chain with cycle and broken-chain detection. Tests cover all format and chain traversal cases.
Backup envelope writing and chain-walking restore
internal/app/backup.go, internal/app/restore.go, internal/app/restore_chain_test.go
Backup writes a base Envelope before the driver dump; on failure it cleans up the temp file. Restore resolves the chain via ResolveChain, optionally truncates at UpTo (returning CodeUser error for unknown IDs), iterates the chain with per-dump progress, and applies Clean only to the base. Tests use fake drivers to assert invocation order, Clean scoping, UpTo truncation, and error cases.
Sync pipeline refactor with Stream and mode gating
internal/app/sync.go, internal/app/sync_test.go, internal/cli/sync.go
Replaces io.Pipe with jobs.NewStream; backup errors propagate via stream.CloseErr and stream.Close() unblocks the writer when restore exits early. Continuous returns an unsupported error; CrossEngine gates on CapCrossEngineSource/CapCrossEngineTarget. CLI adds --cross-engine and --continuous flags. Test uses 8 MiB chunked writes to exercise the bounded stream.
MySQL/MariaDB binlog incremental driver
internal/driver/_mysqlcommon/conn.go, internal/driver/_mysqlcommon/incremental.go, internal/driver/_mysqlcommon/incremental_test.go, internal/driver/mysql/driver.go, internal/driver/mariadb/driver.go
mysqlcommon.Conn gains binlogBinary; NewConn updated accordingly. New incremental.go adds BinlogPosition, CaptureBinlogPosition (tries two SHOW STATUS queries), ValidateBinlogFormat (enforces ROW format), and BackupIncremental (runs fork binlog tool). MySQL and MariaDB drivers set Incremental:true and pass their respective binlog executables.
Postgres WAL incremental driver
internal/driver/postgres/incremental.go, internal/driver/postgres/incremental_test.go
Adds IncrementalBaseInfo, CreateBaseSlot (physical replication slot + WALStart), CaptureBaseEnd (WALEnd via pg_current_wal_lsn()), BackupIncremental (pg_receivewal with PGPASSWORD), and DropSlot. Integration test (build-gated) exercises the full slot lifecycle and asserts non-empty LSN fields.
Cross-engine canonical type system, JSONL emit/consume
internal/app/canonical.go, internal/app/canonical_emit.go, internal/app/canonical_consume.go, internal/app/canonical_test.go
Defines CanonicalType constants, schema structs, MapToNative (engine SQL types with precision/scale), quoteIdent, and placeholder. EmitCanonical streams schema header + table rows as JSONL; ConsumeCanonical reads the header, creates tables, and inserts rows with parameterized SQL. Tests cover all SQL builders, injection neutralization, and JSON round-trips.
CDC scaffold, CLI flags, and docs
internal/app/cdc.go, internal/app/cdc_test.go, internal/cli/backup.go, internal/cli/restore.go, docs/CDC.md, docs/CROSS_ENGINE.md, docs/INCREMENTAL.md, README.md, CHANGELOG.md
RunCDC is a capability-gated (CapCDC) polling scaffold with 10 s heartbeat and JSON state persistence via CDCStateDir. CLI adds --incremental/--base (backup, not-yet-wired guard) and --up-to (restore). New docs cover incremental design, cross-engine type mapping, and CDC scaffold status. README Phase F updated to Partial.

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant Sync
  participant jobs.Stream
  participant BackupGoroutine
  participant Restore

  CLI->>Sync: Sync(opts)
  Sync->>jobs.Stream: NewStream(64 chunks)
  Sync->>BackupGoroutine: go conn.Backup(ctx, stream)
  Sync->>Restore: conn.Restore(ctx, stream)
  BackupGoroutine->>jobs.Stream: Write(chunks) — blocks when full
  Restore->>jobs.Stream: Read(p) — drains chunks
  alt Restore exits early
    Restore-->>Sync: return err
    Sync->>jobs.Stream: Close() — unblocks BackupGoroutine
    BackupGoroutine->>jobs.Stream: Write → io.ErrClosedPipe
  else Backup fails
    BackupGoroutine->>jobs.Stream: CloseErr(bErr)
    Restore->>jobs.Stream: Read → returns bErr after draining
  end
Loading
sequenceDiagram
  participant Restore as app.Restore
  participant Catalog
  participant DumpFile
  participant Driver

  Restore->>Catalog: ResolveChain(dumpID)
  loop walk ParentID (cycle-detected)
    Catalog-->>Restore: []Meta [base, inc1, inc2]
  end
  opt UpTo set
    Restore->>Restore: truncateChain(chain, upTo)
  end
  loop each Meta in chain
    Restore->>DumpFile: Open + ReadEnvelope
    DumpFile-->>Restore: Envelope + body reader
    Restore->>Driver: conn.Restore(opts{Clean: i==0}, body)
  end
Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

  • nixrajput/siphon#1: Established the foundational app.Backup/app.Restore pipeline that this PR extends with SIPH envelopes, chain resolution, and --up-to truncation.
  • nixrajput/siphon#3: Introduced the RequireCapability / CapCDC / CapCrossEngine* capability-gating layer that this PR's CDC and cross-engine gating paths directly consume.
  • nixrajput/siphon#4: Added the shared _mysqlcommon driver infrastructure that this PR extends with binlogBinary, BinlogPosition, and the full binlog incremental implementation.

Poem

🐇 Hop hop, the rabbit lays the pipe,
A bounded Stream, no longer io.Pipe!
SIPH envelopes seal each dump with care,
Chains of incrementals fill the air. 🔗
CDC ticks, the Postgres slot stands tall,
Cross-engine JSONL answers the call. 🐾

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 52.27% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check ❓ Inconclusive The title "Feat/implement phase f" is vague and generic. It uses 'phase f' which lacks specificity about the core change, making it unclear to reviewers scanning history what this PR actually delivers. Use a specific, concise title that highlights the main deliverable, such as 'Implement Phase F advanced transfer: envelope, chain-walking restore, and bounded-buffer sync' or 'Add dump envelopes and incremental restore chain support'.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/implement-phase-f

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 10

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
internal/app/sync.go (1)

97-104: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Preserve the real restore error when forcing stream shutdown.

On Line 98, stream.Close() intentionally unblocks the backup writer; that can make backup return io.ErrClosedPipe. On Lines 101-103, that induced backupErr takes precedence and can mask the real restoreErr from Line 97.

Suggested fix
+import (
+    "errors"
+    "io"
+)
...
 			restoreErr := dstConn.Restore(ctx, driver.RestoreOpts{TargetTables: opt.Tables, Clean: true}, stream)
 			_ = stream.Close() // unblock the backup goroutine's Write immediately if Restore returned early
 			backupErr := <-errCh

+			if restoreErr != nil {
+				if backupErr != nil && !errors.Is(backupErr, io.ErrClosedPipe) {
+					return errors.Join(restoreErr, backupErr)
+				}
+				return restoreErr
+			}
 			if backupErr != nil {
 				return backupErr
 			}
-			return restoreErr
+			return nil
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/app/sync.go` around lines 97 - 104, The code on lines 97-104 has a
logic issue where closing the stream to unblock the backup goroutine can cause
it to return io.ErrClosedPipe, which then masks the actual restore error. Fix
this by checking if restoreErr is not nil and returning it first (since that's
the real operation error), and only return backupErr if restoreErr is nil.
Alternatively, handle the specific case where backupErr is io.ErrClosedPipe (the
expected error from intentionally closing the stream) and suppress it, allowing
the real restoreErr to be returned instead.
🧹 Nitpick comments (3)
internal/driver/postgres/incremental_test.go (1)

14-51: 🧹 Nitpick | 🔵 Trivial

Add integration test for BackupIncremental to catch pg_receivewal invocation regressions.

Currently, no integration test exercises BackupIncremental, so CLI contract changes in the pg_receivewal command invocation (args, env setup) will not be caught. The existing TestIncremental_SlotAndLSNCapture validates only slot lifecycle and base backup operations.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/driver/postgres/incremental_test.go` around lines 14 - 51, Add
integration test coverage for the BackupIncremental method to catch
pg_receivewal command invocation regressions. After the existing slot creation
and base backup operations in TestIncremental_SlotAndLSNCapture (or in a new
complementary test), invoke the BackupIncremental method on the *Conn object to
exercise the WAL streaming via pg_receivewal command, capturing its output and
verifying successful execution. This ensures that CLI contract changes in the
pg_receivewal command arguments and environment setup will be detected during
test execution.
internal/app/cdc.go (2)

88-93: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Use a per-sync state key instead of a global cdc file.

Line 88 hardcodes jobID to "cdc", so different source/target continuous sync pairs will overwrite each other’s resume position file.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/app/cdc.go` around lines 88 - 93, The jobID is hardcoded to the
constant string "cdc" on line 88, causing all source/target sync pairs to share
the same state file and overwrite each other's resume positions. Replace the
hardcoded jobID with a dynamic value that incorporates both opt.From and opt.To
(the source and target) to create a unique key for each distinct sync pair. This
ensures each source/target combination maintains its own separate state file for
proper resume functionality.

91-93: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Only ignore missing-state errors; surface other load failures.

Lines 91-93 currently ignore all loadCDCState errors. A malformed or unreadable state file will silently reset progress instead of failing fast.

Suggested fix
+import "errors"
...
-			if prev, err := loadCDCState(jobID); err == nil {
-				state = prev
-			}
+			if prev, err := loadCDCState(jobID); err == nil {
+				state = prev
+			} else if !errors.Is(err, os.ErrNotExist) {
+				return err
+			}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/app/cdc.go` around lines 91 - 93, The error handling in the
loadCDCState call currently ignores all errors when err is nil, which masks real
failures like malformed or unreadable state files. Modify the error handling
logic in the loadCDCState block to specifically check if the error is a
missing-state error (typically a file-not-found type error) and only ignore that
specific error type. For all other errors returned by loadCDCState, ensure they
are surfaced by logging or returning them rather than silently continuing with
the default state. This way, real state file corruption or access issues will be
caught instead of silently resetting progress.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/CROSS_ENGINE.md`:
- Around line 93-94: The documentation in CROSS_ENGINE.md at the specified lines
overstates the current index handling capabilities. The current implementation
only synthesizes CREATE TABLE IF NOT EXISTS statements and row inserts, but does
not actually recreate indexes. Reword the statement about indexes from claiming
they are created (but not optimized) to accurately reflect that index recreation
is not implemented yet, aligning the documentation with the actual behavior of
the canonical consume path.

In `@internal/app/canonical_consume.go`:
- Around line 19-21: The bufio.Scanner with a 16 MiB buffer limit can cause
token size violations when processing large rows, resulting in mid-stream aborts
and partial replay state. Replace the Scanner-based approach (where sc is
created with NewScanner and Buffer methods are called) with json.Decoder, which
does not impose token size limits on the input stream. This change should be
applied throughout the canonical_consume.go file, including all locations where
the Scanner is currently used for row streaming to ensure consistent behavior
and eliminate the risk of large-row transfer failures.

In `@internal/app/canonical_emit.go`:
- Around line 43-56: The scanned values in the vals slice can contain []byte
which gets marshaled as base64 JSON, but text/JSON/decimal-like values need to
be normalized first. In the loop where the map m is populated from t.Columns,
add normalization logic before assigning each value to m[c.Name] to detect and
transform any []byte values into their appropriate string or typed
representations based on the column type, ensuring that text/JSON/decimal values
are properly converted before being passed to writeJSONL.

In `@internal/app/restore.go`:
- Around line 63-74: The envelope returned from dumps.ReadEnvelope at line 63 is
being ignored (assigned to underscore), which means there is no validation that
the dump's driver matches the current driver before conn.Restore is called with
Clean=true on the first iteration. Capture the envelope instead of ignoring it,
extract the driver information from the envelope/meta, and add a strict driver
match validation before the conn.Restore call at line 74 to ensure the dump is
compatible with the current driver before any destructive operations like clean
are performed.

In `@internal/cli/backup.go`:
- Around line 31-48: The CLI currently accepts the --base flag without enforcing
that it can only be used with the --incremental flag, making the behavior
ambiguous when --base is provided without --incremental. Add a validation check
before or near the existing incremental backup validation that rejects the
combination when baseID is not empty and incremental is false. Return a
user-facing error that explains --base can only be used with --incremental,
similar to the structure of the existing incremental backup error that uses
errs.Error with Op, Code, Cause, and Hint fields.

In `@internal/driver/_mysqlcommon/incremental.go`:
- Around line 101-110: The binlogArgs function does not include SSL/TLS
configuration from the driver.Profile.SSLMode, which means incremental binlog
capture ignores required encryption policies. To fix this, modify the binlogArgs
function signature to accept the binlogBinary parameter (which indicates whether
the binary is mysqlbinlog or mariadb-binlog), create a helper function that maps
p.SSLMode to fork-specific SSL flags (MySQL uses --ssl-mode with
--ssl-ca/--ssl-cert/--ssl-key while MariaDB uses --ssl and
--ssl-verify-server-cert), and append the appropriate SSL flags to the argument
slice before adding the final since.File argument.

In `@internal/driver/postgres/incremental.go`:
- Around line 54-70: The incrementalArgs function contains two issues: the `-D`
argument uses `-` expecting stdout, but pg_receivewal requires an actual
directory path and does not support stdout streaming, so this argument must be
changed to specify a valid directory instead. Additionally, the comment on line
72 incorrectly describes `--no-loop` as controlling WAL segment termination when
it actually controls retry behavior on connection errors, so update the comment
to accurately reflect the flag's true purpose and remove the incorrect
characterization of its function.

In `@internal/jobs/stream_test.go`:
- Around line 177-190: The test TestStream_ConcurrentWriteCloseNoPanic spawns
three goroutines (for Close, Write, and Read operations) inside a loop without
waiting for them to complete, allowing the test to return before the concurrent
operations are actually executed and checked for race conditions. Add a
sync.WaitGroup to synchronize the spawned goroutines: initialize it before the
loop, call Add(3) for each iteration to account for the three goroutines, invoke
Done() at the end of each spawned goroutine function, and call Wait() after the
loop to ensure all goroutines complete before the test returns.

In `@internal/jobs/stream.go`:
- Around line 33-35: The Write method currently enqueues the entire input byte
slice p as a single chunk, which allows large writes to consume unbounded memory
and bypass the capChunks backpressure mechanism. Modify the Write method to
split the incoming p into appropriately sized chunks (respecting the intended
chunk size limit of ~1MB as mentioned in the NewStream documentation) and
enqueue each chunk individually. This ensures that large Write calls are
properly throttled and the buffer backpressure works as designed, rather than
allowing any single Write to circumvent the buffer bounds.
- Around line 72-73: The Read method of the Stream struct must handle
zero-length reads explicitly to comply with the io.Reader contract. Add a check
at the beginning of the Read method (before the overflow check) to return (0,
nil) immediately when len(p) == 0, ensuring that callers probing with empty
buffers do not block on the select statement that waits for data from s.ch or
s.done.

---

Outside diff comments:
In `@internal/app/sync.go`:
- Around line 97-104: The code on lines 97-104 has a logic issue where closing
the stream to unblock the backup goroutine can cause it to return
io.ErrClosedPipe, which then masks the actual restore error. Fix this by
checking if restoreErr is not nil and returning it first (since that's the real
operation error), and only return backupErr if restoreErr is nil. Alternatively,
handle the specific case where backupErr is io.ErrClosedPipe (the expected error
from intentionally closing the stream) and suppress it, allowing the real
restoreErr to be returned instead.

---

Nitpick comments:
In `@internal/app/cdc.go`:
- Around line 88-93: The jobID is hardcoded to the constant string "cdc" on line
88, causing all source/target sync pairs to share the same state file and
overwrite each other's resume positions. Replace the hardcoded jobID with a
dynamic value that incorporates both opt.From and opt.To (the source and target)
to create a unique key for each distinct sync pair. This ensures each
source/target combination maintains its own separate state file for proper
resume functionality.
- Around line 91-93: The error handling in the loadCDCState call currently
ignores all errors when err is nil, which masks real failures like malformed or
unreadable state files. Modify the error handling logic in the loadCDCState
block to specifically check if the error is a missing-state error (typically a
file-not-found type error) and only ignore that specific error type. For all
other errors returned by loadCDCState, ensure they are surfaced by logging or
returning them rather than silently continuing with the default state. This way,
real state file corruption or access issues will be caught instead of silently
resetting progress.

In `@internal/driver/postgres/incremental_test.go`:
- Around line 14-51: Add integration test coverage for the BackupIncremental
method to catch pg_receivewal command invocation regressions. After the existing
slot creation and base backup operations in TestIncremental_SlotAndLSNCapture
(or in a new complementary test), invoke the BackupIncremental method on the
*Conn object to exercise the WAL streaming via pg_receivewal command, capturing
its output and verifying successful execution. This ensures that CLI contract
changes in the pg_receivewal command arguments and environment setup will be
detected during test execution.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0b28c1dc-3011-43cd-a6fd-dc800cb79f7d

📥 Commits

Reviewing files that changed from the base of the PR and between 4135acc and 26f56c7.

📒 Files selected for processing (32)
  • CHANGELOG.md
  • README.md
  • docs/CDC.md
  • docs/CROSS_ENGINE.md
  • docs/INCREMENTAL.md
  • internal/app/backup.go
  • internal/app/canonical.go
  • internal/app/canonical_consume.go
  • internal/app/canonical_emit.go
  • internal/app/canonical_test.go
  • internal/app/cdc.go
  • internal/app/cdc_test.go
  • internal/app/restore.go
  • internal/app/restore_chain_test.go
  • internal/app/sync.go
  • internal/app/sync_test.go
  • internal/cli/backup.go
  • internal/cli/restore.go
  • internal/cli/sync.go
  • internal/driver/_mysqlcommon/conn.go
  • internal/driver/_mysqlcommon/incremental.go
  • internal/driver/_mysqlcommon/incremental_test.go
  • internal/driver/mariadb/driver.go
  • internal/driver/mysql/driver.go
  • internal/driver/postgres/incremental.go
  • internal/driver/postgres/incremental_test.go
  • internal/dumps/chain.go
  • internal/dumps/chain_test.go
  • internal/dumps/envelope.go
  • internal/dumps/envelope_test.go
  • internal/jobs/stream.go
  • internal/jobs/stream_test.go

Comment thread docs/CROSS_ENGINE.md Outdated
Comment thread internal/app/canonical_consume.go Outdated
Comment thread internal/app/canonical_emit.go
Comment thread internal/app/restore.go Outdated
Comment thread internal/cli/backup.go
Comment thread internal/driver/_mysqlcommon/incremental.go Outdated
Comment thread internal/driver/postgres/incremental.go Outdated
Comment thread internal/jobs/stream_test.go
Comment thread internal/jobs/stream.go
Comment thread internal/jobs/stream.go
…s, incremental)

- Rewrite Postgres BackupIncremental to write WAL to a temp dir
  with --endpos (pg_receivewal cannot stream to stdout); fix the
  misleading --no-loop comment (retry, not WAL-end).
- Normalize []byte scanned values to string in canonical emit so
  they JSON round-trip as text instead of base64.
- Replace bufio.Scanner with json.Decoder in ConsumeCanonical to
  remove the 16 MiB token cap that aborted large rows mid-replay.
- Verify each chain dump's envelope driver matches the target
  before any destructive Clean in Restore (ErrIncompatibleEngine).
- Pass fork-specific SSL flags to mysqlbinlog/mariadb-binlog from
  the profile SSLMode.
- Split large Writes into <=1 MiB chunks so Stream's bounded
  channel bounds memory; return (0, nil) for zero-length Reads.
- Join goroutines in the Stream race stress test with a WaitGroup.
- Reject --base without --incremental in the backup command.
- Correct the cross-engine doc: index recreation is not yet
  implemented (data + table structure only).
@nixrajput nixrajput merged commit 95abeb4 into main Jun 21, 2026
5 checks passed
@nixrajput nixrajput deleted the feat/implement-phase-f branch June 21, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant